1. Field of the Invention
The present invention relates to a depth expansion apparatus that combines a plurality of images in different focal positions to expand a depth of focus.
2. Description of the Related Art
To increase a depth of field of an observation system, in general, an aperture diaphragm is restricted (an F value is increased).
There has also been a proposal to configure an observation optical system to include a focus function and enable the observation optical system to focus on a wide range of the depth of field.
On the other hand, various techniques have been proposed for combining a plurality of images in different focal positions to expand a depth of focus.
For example, Japanese Patent Application Laid-Open Publication No. 11-32251 describes a technique for acquiring a plurality of images in different focal positions using a bifocal optical system including birefringent crystal, comparing and selecting luminances of respective pixels of the images in the different focal positions to recombine the images, and obtaining an image with a large depth of focus. The synthesis technique for a depth expanded image described in the publication is a technique for comparing luminances and selecting a pixel value of any one of the plurality of images on the basis of a comparison result.
Japanese Patent Application Laid-Open Publication No. 2001-344599 describes a technique for, before combining a plurality of images in different focal positions, making luminance levels of the respective images uniform and reducing noise.
Further, Japanese Patent Application Laid-Open Publication No. 2004-350070 describes an image processing apparatus that includes a plurality of image pickup devices for photographing images formed by a single photographing optical system and obtains an image formed by combining a plurality of image data photographed by the plurality of image pickup devices. The image processing apparatus includes controlling means for controlling, according to an operation condition of the photographing optical system, positions of the respective image pickup devices to satisfy a condition that depth of field ranges formed by the respective image pickup devices are adjacent to or slightly overlap one another.
Note that Japanese Patent Application Laid-Open Publication No. 8-241396 describes a technique for acquiring, on the basis of the principle that an image cumulatively added up and inputted while a focusing surface is moved in an optical axis direction is a convolution of a response function obtained by projecting a point spread function (PSF) in the optical axis direction and a parallel projected image of an object image, a plurality of images while moving the focusing surface in the optical axis direction and generating, on the basis of the plurality of images corresponding to different focusing surfaces, a plane projected image in a predetermined angle direction along the inside of a fault plane parallel to the optical axis.
Japanese Patent Application Laid-Open Publication No. 2005-49646 and Japanese Patent Application Laid-Open Publication No. 2006-208407 describe an improved stereoscopic display technique of a DFD type (depth-fused 3D) apparatus.
A technique generally used as a method of combining a plurality of images in different focal positions to generate a depth expanded image explained above is a technique for slicing out and combining images within depth.
The technique for slicing out and combining images within depth is explained with reference to FIGS. 1 to 3. FIG. 1 is a diagram for explaining a difference in an image forming position corresponding to an object distance. FIG. 2 is a chart showing a state of a luminance change corresponding to an object distance at the time when two images in different focal positions are sliced out and combined according to the object distance. FIG. 3 is a chart showing a state of a luminance change of a point light source corresponding to a change in an object distance at the time when a distance D0 is set as a focusing position.
A far distance image IMGf and a near distance image IMGn are formed in different positions from an object OBJf at a far distance and an object OBJn at a near distance even if the same image pickup optical system LS is used. In the far distance image IMGf, the far distance object OBJf is a focused image but the near distance object OBJn is a blurred image. Conversely, in the near distance image IMGn, the far distance object OBJf is a blurred image but the near distance object OBJn is a focused image.
Therefore, the far distance object OBJf focused in the far distance image IMGf and the near distance object OBJn focused in the near distance image IMGn are respectively extracted and combined as one image, whereby it is possible to obtain a depth expanded image focused on both of the far distance object OBJf and the near distance object OBJn.
More specifically, as shown in FIG. 2, a focused object distance of the near distance image IMGn is represented as D1 and a focused object distance of the far distance image IMGf is represented as D2 (D2>D1), with a boundary set in an object distance D3 in a middle of D1 and D2, an object present at a nearer distance than D3 is closer to a focus in the near distance image IMGn and an object present at a farther distance than D3 is closer to a focus in the far distance image IMGf. Therefore, in an image, an image portion is extracted from the near distance image IMGf concerning the object at the nearer distance than the distance D3 and an image portion is extracted from the far distance image IMGf concerning the object at the farther distance than the distance D3 and the image portions are combined, whereby a more focused image as a whole is obtained.
Incidentally, for easy understanding, it is assumed that a point light source is imaged. As shown in FIG. 3, a highest luminance value is obtained when the point light source is in a focusing position at the distance D0 and, since magnitude of a blur expands as the point light source moves further away from the focusing position (on any one of the near distance side and the far distance side), the luminance value obtained in the same pixel position falls. The luminance change shown in FIG. 3 is a natural luminance change corresponding to an object distance.